Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Celery tasks refactor #19827

Merged
merged 19 commits into from
Jan 18, 2024
Merged

feat: Celery tasks refactor #19827

merged 19 commits into from
Jan 18, 2024

Conversation

benjackwhite
Copy link
Contributor

Problem

We're trying to improve Celery and one part of this is code-base wise making it clearer and easier to manage.

Changes

  • Refactors all tasks to use shared_task to reduce the circuclar import issue
  • Swaps the scheduled tasks to on_after_finalize as mentioned here
  • Moves almost all tasks into tasks
  • Adds enum for choosing the queue

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

Checked the worker starts up as expected. Will deploy a PR version to test there first.

@github-actions github-actions bot temporarily deployed to pr-fix/celery-refactor January 18, 2024 10:11 Destroyed
@github-actions github-actions bot temporarily deployed to pr-fix/celery-refactor January 18, 2024 10:16 Destroyed
pauldambra
pauldambra approved these changes Jan 18, 2024
View reviewed changes
Copy link
Member

@pauldambra pauldambra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Celery starts and tasks run 🚢

@github-actions github-actions bot temporarily deployed to pr-fix/celery-refactor January 18, 2024 10:23 Destroyed
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 18, 2024 •
edited
Loading

Size Change: 0 B

Total Size: 2 MB

ℹ️ View Unchanged
Filename Size
frontend/dist/toolbar.js 2 MB

compressed-size-action

@github-actions github-actions bot temporarily deployed to pr-fix/celery-refactor January 18, 2024 10:39 Destroyed
webjunkie
webjunkie reviewed Jan 18, 2024
View reviewed changes
posthog/tasks/utils.py Outdated Show resolved Hide resolved
posthog/tasks/tasks.py Show resolved Hide resolved
@github-actions github-actions bot temporarily deployed to pr-fix/celery-refactor January 18, 2024 11:31 Destroyed
@github-actions github-actions bot temporarily deployed to pr-fix/celery-refactor January 18, 2024 11:36 Destroyed
@github-actions github-actions bot temporarily deployed to pr-fix/celery-refactor January 18, 2024 11:53 Destroyed
@github-actions github-actions bot temporarily deployed to pr-fix/celery-refactor January 18, 2024 12:07 Destroyed
@github-actions github-actions bot temporarily deployed to pr-fix/celery-refactor January 18, 2024 12:54 Destroyed
@benjackwhite benjackwhite merged commit 7f174fa into master Jan 18, 2024
93 checks passed
@benjackwhite benjackwhite deleted the fix/celery-refactor branch January 18, 2024 13:28
@sentry-io Sentry.io
Copy link

sentry-io bot commented Jan 18, 2024 •
edited
Loading

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

  • ‼️ CHQueryErrorAllConnectionTriesFailed: Code: 279. posthog.tasks.tasks.clickhouse_errors_count View Issue
  • ‼️ NetworkError: Code: 210. Connection refused (ch-offline.posthog.net:9440) posthog.tasks.tasks.clickhouse_mutation_count View Issue
  • ‼️ SocketTimeoutError: Code: 209. (ch-offline.posthog.net:9440) posthog.tasks.tasks.clickhouse_part_count View Issue
  • ‼️ SocketTimeoutError: Code: 209. (ch-offline.posthog.net:9440) posthog.tasks.tasks.clickhouse_mutation_count View Issue
  • ‼️ SocketTimeoutError: Code: 209. (ch-offline.posthog.net:9440) posthog.tasks.tasks.clickhouse_errors_count View Issue

Did you find this useful? React with a 👍 or 👎

@webjunkie webjunkie mentioned this pull request Jan 25, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pauldambra pauldambra pauldambra approved these changes

@webjunkie webjunkie webjunkie left review comments

Assignees
No one assigned
Labels
deploy
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@benjackwhite @webjunkie @pauldambra